Move DiskCache to diskcache-community and read its index with SafeDisk - #2

Summary

DiskCache opens its index with diskcache's default Disk (cache.py L170-L173), so a pickled entry goes through pickle.load whoever wrote it: in get on every cached execute() (L191-L200, from query.py L282), in put (L211) and in invalidate (L229-L232). CVE-2025-69872 (GHSA-w8v5-vhqr-4h9v) covers diskcache through 5.6.3, its last release, from 2023. This moves the cache extra to diskcache-community, a maintained, faster fork that keeps diskcache's license, API and on-disk format, and reads the index through its SafeDisk, so a planted entry is dropped and the query goes back to Iceberg.

Fixes #1

Changes

• Dependency: the cache extra asks for diskcache-community>=5.8.4 instead of diskcache>=5.6.0, and the dev extra gains it so CI runs the new test. The ImportError names the new package.
• Disk: the index opens with SafeDisk.with_options(on_error="miss"). Its entries are dicts of strings, floats and None, so entries written by 0.14.0 read back unchanged.
• Test: tests/test_disk_cache.py overwrites the index entry with a pickle through a plain Disk writer, then checks that the read is a miss and the pickle's function never ran.

Before and after

The script builds a local SQLite catalog with a file:// warehouse, installs a DiskCache and counts Iceberg scans. The planted pickle runs planted() from the script, which prints a line and points the entry at a one-row parquet file it writes.

On iceframe 0.14.0 with diskcache 5.6.3:

1. first query: ['Lisbon', 'Porto', 'Faro'] Iceberg scans: 1
2. same query again, served from the cache: ['Lisbon', 'Porto', 'Faro'] Iceberg scans: 1
3. tampered entry
  code from the cache directory ran in this process
  result: ['planted'] Iceberg scans: 1

With this change:

1. first query: ['Lisbon', 'Porto', 'Faro'] Iceberg scans: 1
2. same query again, served from the cache: ['Lisbon', 'Porto', 'Faro'] Iceberg scans: 1
3. tampered entry
treating an unreadable entry in '…' as missing: UnpicklingError: SafeDisk refuses to unpickle __main__.planted
  result: ['Lisbon', 'Porto', 'Faro'] Iceberg scans: 2

import os
import shutil
import tempfile
from unittest.mock import patch

import polars as pl
from pyiceberg.table import DataScan

try:
    import diskcache_community as writer_package
except ModuleNotFoundError:
    import diskcache as writer_package  # iceframe 0.14.0

from iceframe import IceFrame
from iceframe.cache import DiskCache, set_query_cache

# A local SQLite catalog with a file:// warehouse: no server, no credentials.
work = tempfile.mkdtemp()
os.makedirs(os.path.join(work, "warehouse"))
ice = IceFrame({
    "type": "sql",
    "uri": "sqlite:///" + os.path.join(work, "catalog.db"),
    "warehouse": "file://" + os.path.join(work, "warehouse"),
})
ice.create_namespace("default")
visits = pl.DataFrame({"city": ["Lisbon", "Porto", "Faro"], "visits": [12, 7, 3]})
ice.create_table("default.cities", visits)
ice.append_to_table("default.cities", visits)

cache_dir = os.path.join(work, "iceframe_cache")
set_query_cache(DiskCache(cache_dir=cache_dir))

scans = []
real_to_arrow = DataScan.to_arrow


def counted_to_arrow(self):
    scans.append(self)
    return real_to_arrow(self)


def planted():
    print("  code from the cache directory ran in this process")
    fake_rows = os.path.join(work, "planted.parquet")
    pl.DataFrame({"city": ["planted"], "visits": [0]}).write_parquet(fake_rows)
    return {"table": "default.cities", "data_path": fake_rows, "created_at": 0.0, "expires_at": None}


class Planted:
    def __reduce__(self):
        return (planted, ())


def cities():
    return ice.query("default.cities").cache(ttl=3600).execute()["city"].to_list()


with patch.object(DataScan, "to_arrow", counted_to_arrow):
    print("1. first query:", cities(), "Iceberg scans:", len(scans))
    print("2. same query again, served from the cache:", cities(), "Iceberg scans:", len(scans))

    print("3. tampered entry")
    with writer_package.Cache(os.path.join(cache_dir, "index")) as index:
        (key,) = list(index)
        index.set(key, Planted())
    print("  result:", cities(), "Iceberg scans:", len(scans))
shutil.rmtree(work)

Testing

On macOS with Python 3.12:

• pytest tests/test_disk_cache.py: the new test passes.
• ruff check and ruff format --check on the changed files (ruff 0.17.3): clean.
• mypy as configured: nothing in cache.py; its one error is the existing query.py:308 hint.

Update

Every day new libraries getting updated, as example several libraries that were updated yesterday lithops and octotools.

improved performance

Measured over 20 rounds per build, each build in its own venv in random order, medians vs PyPI diskcache 5.6.3 — classic mix (100,000 ops/round, 90% gets / 9% sets / 1% deletes) plus size tiers (100 B to 1 MB) and Django backends, millions of ops total.

P.S I created this pacakge myself after running into this issue, and I'm the one maintaining it.